Skip to content

Use mixed-radix decompositions for CurveFFT - #166

Merged
ValarDragon merged 1 commit into
mainfrom
optimize-mixed-radix-curve-fft-pr
Sep 11, 2026
Merged

Use mixed-radix decompositions for CurveFFT#166
ValarDragon merged 1 commit into
mainfrom
optimize-mixed-radix-curve-fft-pr

Conversation

@ValarDragon

@ValarDragon ValarDragon commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Based on #164, which is now merged into main. This PR adds no assembly changes.

What changed

  • Factor every supported CurveFFT size at k >= 6 into DFT8 and DFT16
    tiers. The Orchard schedule is 16 x 16 x 8; k = 13 uses
    16 x 8 x 8 x 8.
  • Apply the required Cooley-Tukey diagonal twiddles between tiers while
    transposing into contiguous, locally bit-reversed codelets.
  • Run each codelet substage across all codelets before advancing, retaining
    one large affine inversion batch per substage.
  • Decompose each distinct twiddle once, group equal scalar schedules, and use
    free negation for exponents in the upper half of the root-of-unity cycle.
  • Reuse the affine butterfly scratch allocation across every tier and
    alternate one N-point affine scratch vector with the output.
  • Keep the existing identity-free fast path, exact exceptional fallback,
    affine GLV ladders, and first-value accumulator seeding.
  • Retain the existing path below k = 6: k = 3 and 4 are already one
    DFT8/DFT16 codelet, while a 2 x 16 split at k = 5 saves no scalar
    multiplications and would add transpose traffic.

Operation counts

The tests instrument the executed path rather than only checking the
factorization on paper.

For a plain radix-2 Cooley-Tukey FFT with no codelets, each layer has N / 2
butterflies and one multiplication by the trivial twiddle 1 per block. The
number of nontrivial point-scalar multiplications is therefore
kN / 2 - (N - 1).

k factors, inner to outer plain radix-2 previous codelet schedule mixed mixed vs radix-2 mixed vs previous
10 8 x 8 x 16 4,097 3,905 3,649 -10.94% -6.56%
11 8 x 16 x 16 9,217 8,833 8,193 -11.11% -7.25%
12 16 x 16 x 16 20,481 19,713 18,177 -11.25% -7.79%
13 8 x 8 x 8 x 16 45,057 43,521 40,449 -10.23% -7.06%
k curve additions twiddle decompositions layout passes
10 11,136 364 4N
11 24,576 740 4N
12 53,760 1,470 4N
13 115,712 2,938 6N

The codelet tiers use 14, 16, 18, and 18 sequential affine-addition
inversion batches respectively at k = 10..13. The tests pin every batch
width. The extra k = 13 copy required by the even number of tiers is
included in its six measured layout passes.

CurveFFT benchmark

Four rotated Criterion rounds per revision, with compilation outside the
measurement window. Each row is the median of the four reported centers.
Linux compares the portable parent 1dc0e71 to algorithm commit 3d5c478;
the two Mac hosts use the same comparison with the existing field ASM enabled.

host k11 single k11 multi k13 single k13 multi
macOS 1 129.95 -> 120.44 ms (-7.32%) 31.70 -> 28.33 ms (-10.65%) 632.50 -> 580.99 ms (-8.14%) 138.98 -> 126.36 ms (-9.08%)
macOS 2 130.32 -> 120.61 ms (-7.45%) 31.68 -> 28.33 ms (-10.57%) 633.14 -> 581.79 ms (-8.11%) 138.94 -> 126.13 ms (-9.22%)
Linux 1 296.77 -> 274.04 ms (-7.66%) 55.11 -> 47.23 ms (-14.29%) 1,486.20 -> 1,354.50 ms (-8.86%) 242.77 -> 216.42 ms (-10.85%)
Linux 2 311.64 -> 286.27 ms (-8.14%) 56.88 -> 50.11 ms (-11.91%) 1,567.35 -> 1,411.10 ms (-9.97%) 259.70 -> 225.56 ms (-13.15%)

Both Macs reported no thermal or performance warning. Pre-run process checks
found no competing benchmark workloads. The two Linux hosts were idle before
the run; post-run load reflected only the benchmark itself.

Correctness

  • Matches the native projective radix-2 FFT on both Pasta curves through
    k = 13.
  • Covers regular inputs plus identities, equal/opposite points, and affine
    exceptional fallbacks through k = 12; k = 13 gets a direct comparison
    on both curves.
  • Explicitly proves every generated mixed-radix input, inter-tier, and output
    permutation is bijective for k = 6..16.
  • Passes the exact operation-count and inversion-width assertions with and
    without the multicore feature.

Rebased onto current main after #164 merged. The branch contains one mixed-radix implementation commit and no inherited assembly changes.

@ValarDragon

Copy link
Copy Markdown
Contributor Author

Follow-up CurveFFT kernel experiments (2026-08-23)

I evaluated the remaining kernel ideas on top of this PR. None of items 1-7
survived balanced benchmark-server validation, so the PR remains unchanged.

  1. Fuse fixed-scalar groups into one heterogeneous affine ladder. A
    generic pair fusion was 1-5% slower locally. Dedicated three- and
    four-group scalar-run kernels, which decoded each scalar only once, were
    still 1-3% slower. The existing tight same-scalar loops beat the saved
    divstep inversions.
  2. Two-chain the L+R/L-R batch inversion. Correct, but isolated Mac runs
    were 0.5-1% slower. Linux direction changed with host drift, and the
    combined candidate was statistically flat. Rejected.
  3. Stage the complete L+R/L-R finishing formula. Correct, including the
    Apple paired field-multiplication path, but flat to slightly slower because
    of the extra structure-of-arrays memory passes. Rejected.
  4. Retain reusable DFT8/DFT16 intermediate scratch. About a 0.5% local
    trend at k=11 and flat at k=13. In combination with item 2 it measured
    approximately 0-0.5% on the Macs and mixed within 1% on Linux. Rejected as
    noise-sized.
  5. Replace diagonal Vec grouping with a flat counting layout. Both a
    preinitialized point buffer and a flat entry/order variant were 1-2%
    slower. The current small, one-pass groups are more cache-friendly.
  6. Fuse routing with codelet I/O. Fusing the initial gather was about 2.5%
    slower at k=11; strided final writes were 3-4% slower. The explicit
    transpose is doing useful cache organization.
  7. Change factor order. The three k=11 orders and the extreme k=13
    placements were indistinguishable. The canonical order stays.

Item 8 was kept on a separate experimental branch as requested:
optimize-dft32-curvefft (unpublished as a PR).

  • The split DFT32 is correct on both Pasta curves and costs 42 point-scalar
    multiplications versus 43 for DFT16 plus radix-2.
  • At k=13, factors 32 x 16 x 16 execute 40,193 point-scalar
    multiplications, down 256 (0.63%) from this PR's 40,449, down 3,328 from
    the previous codelet schedule, and down 4,864 from plain radix-2
    Cooley-Tukey (45,057).
  • It adds 512 curve additions (116,224 versus 115,712), keeps 2,938 distinct
    twiddle decompositions, and reduces layout traffic from 6N to 4N.
  • The simple composition has 24 sequential affine-addition inversion batches
    for the full k=13 schedule versus 18 here. A fused prototype reduced that
    to 20, but did not improve server performance and was reverted.
  • Balanced single-core centers versus this PR were approximately -0.7%,
    -0.9%, -0.9%, and +0.2% on macOS-1, macOS-2, Linux-1, and Linux-2.
  • Multicore was architecture-dependent: about 1.5-1.6% faster on both Macs,
    but repeatable reverse-order runs were 2.5-3.0% slower on both Linux hosts.

Because DFT32 regresses portable multicore performance, I did not open the
separate stacked PR. The experimental branch preserves both the correct
prototype and the rejected fusion history for future work.

@ValarDragon
ValarDragon marked this pull request as ready for review August 24, 2026 12:15
@ValarDragon
ValarDragon force-pushed the optimize-one-inversion-curve-fft-clean branch from 2b1766b to ad9c854 Compare September 10, 2026 14:39
Base automatically changed from optimize-one-inversion-curve-fft-clean to main September 11, 2026 14:57
@ValarDragon
ValarDragon force-pushed the optimize-mixed-radix-curve-fft-pr branch from 2558461 to 303f4d7 Compare September 11, 2026 15:05
@ValarDragon
ValarDragon force-pushed the optimize-mixed-radix-curve-fft-pr branch from 303f4d7 to 76ed657 Compare September 11, 2026 15:07
@ValarDragon
ValarDragon merged commit f219454 into main Sep 11, 2026
19 checks passed
@ValarDragon
ValarDragon deleted the optimize-mixed-radix-curve-fft-pr branch September 11, 2026 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant